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Abstract 

This report examines effects of structural and process features of 
professional development programs on teachers' knowledge, practice and 
efficacy. It is based on four recent (2002-2003) studies undertaken through 
the Australian G ovemment Q uality T eacher Programme, designed to 
enhance teacher guality. The total data set for the survey study includes 3,250 
teachers who had participated in eighty individual professional developmeniJ 
activities within these studies. T eachers were surveyed at least three months 
after participating in an activity, which provided them with the opportunity to 
gauge the impact of programs on their practice. T o investigate factors 
affecting impact, a theoretical model was developed based on recent research 
into the characteristics of effective professional development and tested using 
blockwise regression analysis. The model included contextual factors (e.g., 
school support), structure features of programs (e.g. ,length), process 
features (e.g., emphasis on content; active learning; examination of student 
work; feedback; follow-up), a mediating variable (level of professional 
community generated), and four outcome measures (knowledge; practice; 
student learning and efficacy). Consistent significant direct effects were found 


1 In thisarticle, "programme" refers to the Australian Government Quality Teacher Programme 
(AGQTP). Projects funded under AG QTP consist of either State and Territory professional learning 
projeds or national strategic projeds (see p. 3 for further description). State and T erritory professional 
learning projeds develop and deliver professional learning activities for teachers, either on a State or 
T erritory - wide basis or for individual education authorities within that State or T erritory. 


E ducation Policy A nalysis A rdiives V ol. 1 3 N o. 1 0 


2 


across the four studies for the impact of content focus, active learning, and 
follow-up on knowledge and professional community. Feedback was rarely 
incorporated into program design. Impact on efficacy was strongly related to 
the perceived impact of activities on teachers' practice and student learning 
outcomes. 


Introduction 

Professional development for teachers is now recognised as a vital component of policies 
to enhance the guality of teaching and learning in our schools. Conseguently, there is 
increased interest in research that identifies features of effective professional learning. 
Considerable funds are allocated to a wide variety of professional development programs 
from a variety of sources. As investment increases, policy makers are increasingly asking 
for evidence about its effects not only on classroom practice, but also on student learning 
outcomes. They are also looking for research that can pide them in desiping programs 
that are more likely to lead to sipificant and sustained improvement in students' 
opportunities to learn. 

There is a need, therefore, for more sophisticated methods for evaluating professional 
development programs, with the capacity to meet these information needs. In the not 
too distant past, when many professional development courses placed teachers in the 
role of an audience, guestionnaires distributed at the door as teachers left sufficed. 
Strategies for professional development have now become much more complex, long 
term and embedded in schools. Major funds may be allocated, for example, to training 
school- based staff developers and providing them with time release, developing 
curriculum support materials, time release, and on-line learning. 

The kinds of guestions that evaluators now need to answer are much more penetrating 
than guestions such as "What did you learn from the workshop?" They are guestions 
about propam logic and the presumed links between professional learning strategies, and 
changes in teacher knowledge, classroom practices and student outcomes. These 
guestions call for large-scale studies with the capacity to tests these relationships across 
large numbers of different professional development propams. 

Purposes of the study 

This paper reports on the effects of structural and process features of professional 
development propams on teachers' knowledge, practice and efficacy. It is based on four 
evaluation studies recently completed by ACER from 2001 to 2003 under the Australian 
GovemmentQualityTeacher Programme (AG Q TP), TheAGQTP is intended to 
update and improve teachers' skills and understanding and enhance the status of 
teaching. It currently has two elements: 

• State and T erritoiy professional learning projects, which have been contracted 
with government and non- government school education authorities to develop 
and deliver professional learning activities to school teachers; and 

• Strategic national projects focussing on research, investigation and development of 
range of issues relating to teacher guality and school leadership. 



Ingvarson et al.: Impact of Professional D evdopment Programs 


3 


The AG QTP, (which was formerly titled the Q uality T eacher Programme over 2000- 
2003), received total funding of $A76.8 million over 2000-2003, with a further $A82,4 
million provided for 2003- 2005. Of these amounts, $97,2 million has so far been 
allocated to professional learning projects. 

Most of the State and T erritoiy professional learning projects conducted over 2000-2003, 
undertook an evaluation or review of activities. The four studies included in this paper 
are: 


■ Evaluations of the Australian G ovemment Q uality T eacher Program 
professional learning projects, as implemented by education authorities in 
three states (hereafter Projects A, B and C). (Note: 'Education authority' 
refers to an organisation which represents, or is the governing body, of a 
system of schools, either government. Catholic or Independent schools. Eor 
example, one of the education authorities included in this study is the Catholic 
Education Commission in the state of Victoria). 

■ An AG Q TP strategic national project involving a major research study of ten 
professional development activities, conducted by ACER and titled. 

Investigating the links between professional development and student learning outcomes 
(hereafter Project D ), 

While this paper utilises data and analyses from these studies, it should not be seen as 
providing a summary or overview of their findings. The findings from some of these 
studies, such as the AG QTP project. Investigating the links between professional development and 
student learning outcomes, have not yet been released. 

The central purpose of the evaluation was to examine the effectiveness and quality of the 
professional development initiatives funded under the NSW QTP, and the extent to 
which the intended outcomes of the NSW Q TP had been achieved. 


Approach to Evaluating Professional Development Activities 

Cross- program analysis 

In each of these evaluation studies, data was gathered from a number of professional 
development programs. In evaluating one state program, for example, data was gathered 
from over 40 different professional development programs and 1731 teacher- 
participants. Australian Government pidelines required that state programs target 
experienced primaiy and secondary teachers (more than ten years) and focus on hteracy, 
mathematics, science, technology and vocational education. The professional 
development programs utihsed a range of delivery modes, including: 

• W orkplace learning through action research, coaching and mentoring; 

• Institutional learning to facilitate understanding of research findings and best 
practice; 

• Q nline learning; 

• Participation in formal award programs; 

• C onferences and seminars. 
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Individual professional development programs in each state used a combination of 
profession^ development strategies, such as workshops, accredited courses, online 
learning, mentoring, and action learning. Some projects were conducted over extended 
periods of time, others were conducted as single- session workshops, and others used a 
combination of organisational structures, such as an initial workshop sessions prior to a 
school-based project. Some projects were conducted in phased stages, and some 
included related sub- projects. (Appendix C provides a profile of one of the professional 
development programs (Program "G "), summarising the main features of its program 
logic.) 

States and Territories receiving Australian Government funding were also expected to 
commission independent evaluations of their programs that addressed the following 
issues: 

a) the nature and extent of the participation of teachers from specified target 
groups; 

b) participant satisfaction with the guality of the programs; 

c) the extent to which the program achieved its outcomes and objectives; 

d) the contribution of the program to improving and expanding pedagogy; 

e) the contribution of the program to improving student learning outcomes in 
schools; 

f) the extent to which the program enhanced the status of teaching. 

While it was feasible to address some of these issues in the time available for the 
evaluation studies, others, such as effects on student outcomes and the status of 
teaching, were more problematic. 0 ur evaluations (Projects A , B, and C) relied mainly 
on teacher self reports. (Project D mentioned above did include separate measures of 
student learning outcomes before and after the 10 professional development programs). 
While we have some confidence in teachers' self reports of the effects of professional 
development programs on their knowledge and practice, we have less confidence in 
teachers' self reports of the impact of these programs on their students' learning 
outcomes. 

In all, data was gathered from a total of 3250 teachers who had participated in over 
eighty different professional programs. These evaluation studies provided a unigue 
opportunity to conduct research looking at the differential impact of a wide range of 
profession^ development strategies. 

Participants in each of these programs were invited to complete a common survey 
instrument, which asked them to describe both the processes of learning that they had 
experienced and the impact of these programs on their knowledge, practice, sense of 
efficacy, and their students' learning. The survey also asked participants about the 
impact of the programs on the nature and extent of collaborative work amongst 
colleapes in their schools. The extent to which programs strengthened, or integrated 
with professional community activity turned out to be a significant predictor of impact. 

As might be expected, there were significant differences between programs in the mode 
of delivery and in the extent to which teachers reported that programs had influenced 
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their practice and benefited their students. These differences opened up the possibility 
for cross- pro gram analyses that might: 

a) increase understanding of those features of program design and delivery that 
might explain variation in impact; 

b) identify school level factors that influence or mediate the outcome of the 
programs. 

Another feature of these studies was that teachers were surveyed at least three months 
after participating in a program, which provided them with a better basis on which to 
gauge the impact that programs had had on their practice. However, this delay was at 
some cost to response rates to our mailed surveys, which averaged around 50%. 

Research-based conceptual hamework 

These analyses called for the development of a conceptual framework to pide the 
evaluation. The approach to evaluation in each of the four studies was based on the 
theoretical framework, shown in Fipre Lit presents a model of the main features of 
programs that might explain variation in their reported impact. The framework was 
based on a review of recent research into the characteristics of effective professional 
development programs (Kennedy, 1998; Wilson & Berne, 1998; G aret et al., 2001; Sykes, 
2002; Cohen & Hill, 2000, Hawley & Valli, 1999; Guskey & Sparks, 2002; Loucks- 
Horsley et al. 1998; Supovitz, 2001). This research has become increasingly sophisticated 
over recent years (Ingvarson, 2002) and provided a firm foundation on which to develop 
a model to account for the relative differences in the effectiveness of professional 
development programs. 

Fipre 1 distinpishes four, linked, types of impact resulting from PD propams. These 
include impact on teachers' knowledge and practice, student learning and teacher 
efficacy. The model also includes backpound (control) variables, structural features, 
such as the duration of the propam and opportunity to learn features, such as "active 
learning", or "follow up" . (D etails of how these variables were measured are provided 
below.) 


Measures 

Control variables 

Control variables included: teacher gender; experience, measured as number of years 
teaching; school sector; and school support for professional development. School support 
was measured by asking teachers the extent to which they apeed or disapeed with the 
following statements: 

• The leaders at my school actively support and encourage all staff to take 
part in professional development. 

• Insufficient time is available in my school to support teachers' professional 
learning 

• Follow up support for professional development is available within my 
school. 

• T eachers at my school work collaboratively to resolve teaching and 
learning issues. 
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Stnictural features 


Duration includes two slightly different, but related, aspects: oontad hours and time span. To 
measure contact hours, teachers were asked to indicate the total number of hours they 
spent in activities related to the professional development program (1 = less than 10; 2 = 
10-20 hours; 3 = 20+ hours; and 4 = 50+ hours). Nearly 40 % of the programs in this 
sample were over 20 hours in length. 

To measure time span, teachers were asked to indicate the total time the professional 
development activity covered; from 1 = less than one week; 2 = one month or less; 3 = six 
months or less; and 4 = more than six months. Roughly 25% of programs were one week 
or less, while nearly 35% of teachers reported that their programs lasted over six months. 
Collective participation was measured by whether more than one teacher from a school 
was reguired to participate in the program, (This measure was not included in all four 
studies and has been deleted from this analysis. In studies where it was included, it did not 
have a significant impact). 

0 ppoitunity to learn 

An important feature of the model in Figure 1 is that it makes a distinction between design 
and implementation features of professional development programs. 0 ne of our first steps 
in each of the four evaluation studies reported here was to conduct analyses of the logic and 
theory of action underpinning professional development programs. We worked in close 
collaboration with program designers to identify the essential and critical features of the 
professional development models they were using. This included identifying the 
assumptions about teacher learning on which their models were based, and teasing out the 
theory of action underlying their programs (how the features of the proposed model linked to 
each other and how they would lead to change). 

The purpose was to identify the key design and process features of a professional 
development program and how it was meant to 'work'. This task was not always as 
straightforward as it might seem, as some program designers had not been ask^ to 
articulate the theory of action underpinning their programs before. D esigners of 
professional development activities select from a wide range of strategies to promote 
professional learning. They often describe the strategies they have chosen in ways that are 
not necessarily helpful as measures for research purposes. They may use terms such as, 
'hands on', 'action research', 'workshops', 'training sessions', and 'case methods'. What 
these terms actually mean in terms of teacher learning processes is not always clear, T o 
make the research task even more complex, designers often say they use a large number of 
these strategies in the same activity. So we found it difficult to gain useful measures of 
actual teacher learning processes by asking program designers about the strategies that 
characterised their activities. 

In working with program designers, ACER staff drew extensively from recent research on 
the critical features of effective professional development programs (Hawley & Valh, 1999). 
Use was made of other researchers (e.g. Loucks-Horstey et al„ 1998; Sykes, 2002) who 
provide useful pides to the major types of strategies used to promote professional learning, 
Heller et al. (2003) and Killion (2003) provide approaches that help to identify the logic 
underlying programs and how the pieces fit together to promote effective teacher learning. 

The evaluation team used this research to create an instrument for measuring the extent to 
which teachers actually experienced certain types of learning opportunities during the 
professional development program. In developing this instrument (T he 0 uality of Professional 
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L earning Index ) we used our review of the research literature to identify a number of 
characteristics of effective professional development. These included: 

• Content focus 

• Followup 

• Active learning 

• Feedback 

• Collaborative examination of student work 
E ach of these measures is described briefly below. 

Content focus 

Recent research (Kennedy, 1998) indicates the importance of what teachers have the 
opportunity to learn during a professional development activity - this research suggests 
that the substance of what teachers learn is more important than the form or structure of 
the program (e.g. whether programs are school- based or not, collaboratively planned or 
not, extended over time, etc.). In summary, this research indicates that professional 
learning is more likely to improve student learning outcomes if it increases teachers' 
understanding of the content they teach, how students learn that content and how to 
represent and convey that content in meaningful ways (Cohen & Hill, 2000). 

T 0 measure content focus, teachers were asked about the emphasis given to four aspects of 
content: content or subject knowledge, knowledge of how students learn content, 
knowledge of methods of teaching content and models to illustrate those methods of 
teaching of that content.^ Average scores for content focus across the four Programs were 
about three on the four- point scale (Cronbach Alpha = 0.79). 

Active learning 

Recent research confirms the importance of importance of teachers being actively engaged 
in their own learning, but it is the nature of this engagement that seems to matter as much, 
if not more, than the level. Effective professional development programs draw teachers 
into an analysis of their current practice in relation to professional standards for good 
practice. They also draw teachers into close comparison of what their students are learning 
in relation to what students of that age and circumstance are capable of learning, 

T 0 measure active learning, teachers were asked about the extent to which a program 
engaged them actively in reflecting on their practice, in identifying specific areas of their 
practice that they needed to develop, and gave them opportunities to test new teaching 
practices. Average scores for active learning across the Programs were slightly above three on 
the four-point scale (Cronbach Alphas 0.79). 

Feedback 

Eeedback on practice has long been recognised as a vital reguirement for professional 
development programs that aim to help teachers develop new skills and integrate them into 
their practice (Joyce & Showers, 1982), Effective integration of new skills reguires 
programs to have a clear theoretical foundation supported by research, modelling in real 
settings, and opportunities to practice the new skills and receive feedback from a coach or 
supporting teacher. Most of the programs we have evaluated recently aimed to help 


2 T 0 measure content focus, an index was developed based on four items. T eachers responded to 
these items on a four- point scale (1 = no emphasis, 2=minor emphasis, 3= moderate emphasis, 4 = 
major emphasis). The scores of each of these items were averaged to give a measure of content 
focus. A similar process was used to construct all measures 
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teachers learn new skills. However, we found that few participants actually received 
assistance and feedback in their classrooms during the critical and difficult implementation 
phase when they were trying out new practices. 

T 0 measure feedback, teachers were asked about the number of times they received 
feedback on their teaching from other teachers or people involved in the program; and the 
number of times their teaching was observed by others involved in the program (e.g. from 
a mentor, or in a team teaching situation). Average scores for feedback across the Programs 
were low - about 1.5 on the four- point scale (Cronbach Alpha = 0.86). 

Collaborative examination of student work 

Effective professional development programs lead teachers to examine their students' work 
in relation to external reference points or standards. Hawley and Valli's (1999) review of 
research rates this feature as a critical component of effective professional learning 
programs. It has become clear over recent years that teachers gain a great deal of valuable 
learning from opportunities to examine student work in collaboration with colleagues - 
especially their own students' work, and in relation to standards for what students should 
know and be able to do. Collaborative analyses of student work opens up many avenues 
for teachers to de- privatise their practice and learn from each other. It also leads to deeper 
understanding of student learning outcomes and greater discrimination about what counts 
as meeting those objectives. 

To measure oollaborative examination of student work , we developed an index based on the 
extent to which teachers said they received opportunities to collaborate with colleapes in 
examining their own students' work as well as that of other teachers. Average scores for 
oollaborative ec amination of student work across the Programs were about 2.5 on the four- point 
scale (Cronbach Alpha = 0.84). 

Follow-up 

Follow-up support to teachers during the implementation phase of change has long been 
identified as an important feature of more effective programs (Fullan, 1982). Perhaps the 
strongest criticism of many professional development programs over the years has been the 
lack of built in provision for 'at the elbow' support for teachers in their classrooms as they 
apply new ideas and skills (Huberman & Miles, 1984). 

To measure follow-up we developed an index based on the extent to which teachers reported 
that a program provided time for follow-up and ongoing assistance in their school or 
classroom to help them implement changes advocated in the program and opportunities to 
practice their new learning. Average scores for follow-up across the Programs were about 
2.5 on the four- point scale (Cronbach Alpha = 0.79). Factor analysis confirmed the scales 
used to measure the five opportunity to loam constructs described above. D etails about the 
psychometric properties of these opportunity to loam variables can be provided on reguest. 

Appendix A presents means and confidence intervals for each of the opportunity to learn 
variables for 26 professional development programs within one of the state- level programs. 
The findings show significant variation across the 26 programs in this state program. 
Readers will notice that some programs such as "G " are consistently rated highly across the 
process measures and others such as "P" are consistently rated low. 

Mediating variables 

Many professional development programs aim to strengthen professional community in 
schools in order to enhance the impact of their programs on classroom practice. 
Therefore, professional community was included in our model as a mediating variable. In 
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measuring, professional community, teachers were asked to respond to items such as: 


• T eachers at my school discuss teaching and learning more with their colleapes 

• Teachers have increased their collaboration in planning, teaching and assessment 
activities 

• I have passed ideas I learned from the program on to other teachers in my school. 

Once again, there was significant variation across programs in the extent to which they 
stimulated professional community activity in schools (Cronbach Alpha 0.96). 


Measures of impact 

In order to conduct research across several state- level programs, based on the conceptual 
model in Fipre 1, we developed an approach to conceptualising and identifying outcomes 
of professional development programs based on standards for effective teaching 
(Ingvarson, 1998; 2002). We arped that the guality of impact of a program should not only 
be measured in terms of whether it meets the developers' objectives, but also in terms of 
the extent to which the program moves teachers' practices towards those associated with 
research- based standards for effective teaching (Ingvarson, 1998; 2002). (The objectives 
and standards may be much the same, but we found this is not necessarily the case) 

Four aspects of impad were selected: impact on teachers' knowledge impact on teachers' 
practice; impact on student learning outcomes; and, impact on teacher efficacy. These variables are 
described in more detail below. 

Knowledge. Teachers were asked to indicate the extent to which their participation in 
the professional development program had led to increased knowledge of: the content 
they teach, teaching and learning strategies appropriate to the content they teach, how 
students learn the content, individual differences amongst students and how to cater for 
their needs, how to link assessment into the teaching and learning cycle, classroom 
organisation and management. Teachers reported their responses to the individual items 
on a four-point scale from strongly agree to strongly disagree. The same scale applied 
to all four impact measures (Cronbach Alpha = 0.92). 

Practice. Teachers were asked whether, as a result of their participation in the 
professional development program, they now: 

■ make clearer links between their teaching go als and classro om activities; 

■ manage classro om structures and activities more effectively; 

■ use more effective teaching and learning strategies appropriate to the content that they 
teach; 

■ use more effective teaching and learning strategies appropriate to the classro om 
context; 

■ use teaching and learning strategies that are more challenging and engaging; 

■ are better able to meet the individual learning needs of their students; 

■ link assessment into the teaching and learning cycle more effectively; 

■ provide more effective feedback to their students to support their learning; 

■ engage students in higher order thinking; 
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■ access and use materials and resources more effectively. (Cronbach Alpha = 0.93) 


Student learning outcomes. For example, teachers are asked whether, as a result of the 
PD program, their students now: 

■ have fewer difficulties in understanding what they are being taught; 

■ are learning more purpo sefully; 

■ are more actively engaged in learning activities; 

■ demonstrate enhanced learning outcomes; 

■ access and use materials and resources more effectively. (Cronbach Alpha = 0.95) 

(Precise wording of these items varied slightly from program to program to match specific 
goals) 

Teacher efficacy. Teachers are asked about the extent to which they agree or disagree 
with the following statements as a result of the PD program: 

■ My ability to meet the learning needs of my students has expanded 

■ My confidence in teaching [subject] has increased. (Cronbach Alpha = 0.85) 

Factor analyses confirmed that the four impact measures had strong scale characteristics 
and were sensitive to differences across programs. D etails about the psychometric 
properties of the impact variables can be provided on reguest. 


Comparisons of PD programs in terms of impact 

Appendix B presents means and confidence intervals for each of the impact measures 
across the same 26 professional development programs as in Appendix A. The findings 
show the extent of variation across the 26 programs. Readers will note once more that 
some programs (eg "G ") are consistently rated high across the measures while others are 
consistently rated low (e.g. "JJ"). 

Appendix C provides a summary of the professional development strategy for Program 
"G " . It is clear from this summary how the designers have incorporated most of the 
features characteristic of programs consistent with research on effective professional 
learning. E specially noteworthy is the inclusion of feedback using videos to promote 
reflection and follow up visits by the facilitator. 


Analysis 

Separate analyses of the data were conducted for each of the four state-level programs. 
This paper brings the findings of these four independent studies together. As a first step in 
each analysis, a blockwise regression analysis was conducted using the SPSS PC procedure 
regression. This procedure is based upon a least-sguares algorithm to estimate the strength 
of the linear relationship between the dependent variable and a set of independent 
variables. The order in which these variables were entered into the eguation was 
determined by the theory underlying the research (as summarised in Fipre 1). There were 
six control, or background (exogenous), variables in this model, and three blocks of 
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intervening (endogenous) variables: structural features, learning processes and professional 
community. (An example of the results a full analysis of a state- level program is shown in 
Appendix D ), For the purposes of this paper, however, these full tables have been limited 
to summarising the regression coefficients for the structural and opportunity to learn 
variables. 

Table 1 shows the standardised regression coefficients and significance levels for each of 
the structural and opportunity to learn variables in the model, across the four state-level 
programs. Significant relationships are indicated in bold type. The use of standardised co- 
efficients permits easy comparison of the strength of associations within the model. For 
example, a standardised beta coefficient of 0.27 is three times as strong in its effect as one 
of 0.09. When examining these effects it is important to remember that they are net of the 
effects of other variables in the model. The regression analysis thus shows the unigue 
contribution that each variable makes to changes in the dependent variable. 


Findings 

T able 1 provides an amalgamation of findings across the four evaluation studies.^ These 
four studies, in effect, are replications of the same experiment. When findings are 
replicated across several studies, they can be reported with greater confidence. This 
amalgamation of findings has enabled a more rigorously testing of our conceptual model 
(of factors affecting the impact of professional development programs). Using this model, 
we have investigated relationships between process features of professional development 
programs, such as active learning and follow-up, and outcome measures, such as teacher 
practice and efficacy, across the four studies. 

Table 1 shows the proportion of variance explained by the model (R^) across the four 
studies. Although there is some variation in the explained variance, overall there is a degree 
of consistency across the studies. For example, the full model accounted for around 48% 
of the variance in the dependent variable tocher efficacy in Program A, 51 % in Program B, 
45% in Program C and 69% in Program D - which meant that several features in our 
model were reasonably good predictors of whether teachers would rate a professional 
development program as effective. 

One of the main findings from this study is that, among all the variables in the model 
(Fipre 1), the block of opportunity to learn or process variables had the largest effect on 
individual program outcomes. When we look at the reported impact of these variables 
across the four studies, as summarised in T able 1, a reasonably consistent pattern emerges. 


3 Reports of each evaluation study can be found at 
http:/ / www.acer.edu.au/ research/ programs/ teaching.html 
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Table 1 


Relationships between PD characteristics & impact measures across four Programs 


PROGRAM A 

Prof Cmty 

Knowledge ' 

Practice 


Student outcome Efficacy 


Total hrs for activity 

0.01 

0.68 

0.04 

0.30 

0.03 

0.44 

0.03 

0.53 

0.00 

0.96 

Span of time 

0.15 

0.00 

-0.05 

0.20 

-0.08 

0.07 

-0.02 

0.73 

0.04 

0.40 

Content 

0.15 

0.00 

0.19 

0.00 

1 

0.20 

0.00 

0.16 

0.00 

0.15 

0.00 

Active 

0.00 

0.90 

0.17 

0.00 

1 

0.10 

0.04 

-0.04 

0.48 

0.13 

0.00 

Follow up 

0.25 

0.00 

0.09 

0.03 

0.06 

0.15 

0.04 

0.43 

0.19 

0.00 

Collabora.tion 

0.21 

0.00 

0.08 

0.05 

0.00 

0.95 

0.00 

0.93 

-0.15 

0.00 

Feedback 

0.00 

0.93 

-0.05 

0.11 


-0.02 

0.62 

0.12 

0.00 

0.09 

0.01 

Prof, community 



0.10 

0.01 


0.27 

0.00 

0.00 

0.96 

0.11 

0.01 

Knowledge 






0.17 

0.00 

0.29 

0.00 

0.12 

0.00 

Practice 








0.23 

0.00 

0.10 

0.02 

Student outcomes 










0.20 

0.00 

R squared (adjusted) 

0.36 


0.23 



0.34 


0.35 


0.48 


PROGRAM B 



Total hrs for activity 

-0.02 

0.58 

-0.04 

0.19 

-0.03 

0.20 

-0.07 

0.02 

0.03 

0.39 

Span of time 

0.01 

0.66 

0.04 

0.22 

0.06 

0.03 

0.03 

0.37 

-0.06 

0.06 

Content 

0.11 

0.01 

0.39 

0.00 

1 

0.09 

0.01 

0.13 

0.00 

O.B 

0.00 

Active 

0.00 

0.91 

0.11 

0.01 


0.12 

0.00 

0.03 

0.48 

0.20 

0.00 

Follow up 

0.29 

0.00 

0.09 

0.03 

0.09 

0.01 

0.08 

0.02 

0.03 

0.40 

Collaboration 

0.23 

0.00 

0.05 

0.22 


-0.10 

0.00 

-0.04 

0.24 

-0.12 

0.00 

Feedback 

0.16 

0.00 

0.03 

0.26 


0.02 

0.44 

-0.06 

0.05 

0.09 

0.00 

Prof, community 



0.12 

0.00 

1 

0.15 

0.00 

0.11 

0.00 

0.07 

0.03 

Knowledge 






0.56 

0.00 

-0.01 

0.81 

0.00 

0.98 

Practice 








0.56 

0.00 

0.18 

0.00 

Student outcomes 










0.33 

0.00 

R squared (adjusted) 

0.36 


0.39 



0.60 


0.51 


0.51 


PROGRAM C 












Total hrs for activity 

-0.06 

0.48 

0.01 

0.91 


0.07 

0.30 

0.12 

0.08 

0.08 

0.32 

Span of time 

0.12 

0.13 

0.01 

0.90 

1 

0.09 

0.17 

0.15 

0.03 

0.04 

0.65 

Content 

-0.01 

0.87 

0.21 

0.00 

1 

0.00 

0.97 

-0.01 

0.88 

0.09 

0.24 

Active 

0.00 

0.95 

0.27 

0.00 

1 

0.18 

0.01 

0.08 

0.24 

0.05 

0.55 

Follow up 

0.27 

0.00 

0.15 

0.04 

0.14 

0.06 

0.08 

0.29 

0.01 

0.88 

Collaboration 

0.33 

0.00 

0.08 

0.28 

1 

0.07 

0.29 

0.12 

0.11 

0.10 

0.23 

Feedback 

0.15 

0.03 

0.06 

0.32 


-0.08 

0.17 

0.09 

0.15 

0.06 

0.42 

Prof, community 



0.24 

0.00 

1 

0.17 

0.02 

-0.06 

0.39 

0.12 

0.17 

Knowledge 






0.54 

0.00 

0.28 

0.00 

0.09 

0.39 

Practice 








0.48 

0.00 

0.16 

0.14 

Student outcomes 










0.34 

0.00 

R squared (adjusted) 

0.36 


0.51 

0.59 

0.58 


0.45 


PROGRAM D 




Total hrs for activity 

-0.08 

0.15 

-0.04 

0.52 


0.06 

0.14 

-0.06 

0.25 

0.03 

0.50 

Span of time for 












activity 

0.12 

0.01 

0.09 

0.07 


0.00 

0.93 

0.00 

0.92 

-0.02 

0.70 

Content 

-0.05 

0.47 

0.33 

0.00 

1 

0.00 

0.92 

0.09 

0.19 

0.01 

0.85 

Active 

-0.08 

0.20 

0.08 

0.17 

0.15 

0.00 

0.02 

0.72 

0.16 

0.00 

Follow up 

0.50 

0.00 

0.23 

0.00 

1 

0.02 

0.68 

0.10 

0.20 

0.03 

0.67 

Collaboration 

0.31 

0.00 

0.01 

0.87 

-0.04 

0.44 

0.02 

0.72 

-0.03 

0.58 

Feedback 

0.08 

0.17 

0.08 

0.14 

-0.01 

0.76 

0.11 

0.05 

-0.03 

0.49 

Prof, community 



0.14 

0.03 

0.09 

0.05 

0.11 

0.09 

-0.01 

0.87 

Knowledge 






0.72 

0.00 

-0.16 

0.08 

0.02 

0.80 

Practice 








0.65 

0.00 

0.26 

0.00 

Student outcomes 










0.52 

0.00 

R sq. (adjusted) 

0.49 


0.51 



0.76 


0.58 


0.69 
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The most important influence on reported impact onknowledge, perhaps unsurprisingly, is 

the extent to which the program focused on content, as defined above. Across all four state- 

level Programs, the relationship between content focus and impact on knowledge is strong 

(Program A ,19, Program B ,39, Program C .21, Program D ,33). The relationship between 

follow-up and reported impact on knowledge is also significant 

across all four studies and the opportunity for active learning is significant in three of the 

studies. 

The most important influence on reported impact on practice, apart from knowledge is 
entered into the eguation, is the extent to which individual programs provide many 
opportunities for adive learning and refledion on pradice. Across all four state- level Programs, 
the relationship between active learning and impact on practice is significant (Program A 
.10, Program B ,12, Program C .18, Program D ,15). 

When we look at reported impact on knowledge and practice together, the significance of 
professional community as a mediating variable becomes apparent. Table 1 indicates that 
relationships between professional community and the reported level of impact on knowledge 
and practice were significant in all four studies. The extent to which a professional 
development program influences knowledge and practice, as reported by teachers, is 
enhanced by the extent to which that program also strengthens the level of professional 
community in the school; that is, the extent to which it increases opportunities for teachers to 
talk about the specifics of their teaching practice and student learning, share ideas and 
support each other as they attempt to implement ideas from the professional development 
program. The extent to which programs influenced the level of professional community 
activity was enhanced to the extent that their designers built in active learning processes, 
follow up and opportunities for collaborative examination of student work. 

Table 1 shows some interesting features when we look at reported impact on student learning 
outcomes and efficacy. The level of content focus continues to be important in Programs A 
and B, the level of active learning is strongly related to teacher efficacy in three of the 
Programs and, for Programs A and D , feedback starts to have a significant effects over and 
above the strong effects of knowledge and practice. The fact that the influence of active 
learning carries through to influence teacher efficacy, net of the effects of other variables in 
the model, is particularly noteworthy. It suggests that this feature is having a pervasive and 
generative influence on factors that increase teachers' confidence and ability to meet 
student needs than making specific changes in practice alone. The opportunity to learn 
features in this model had a significant direct effect on teacher knowledge. As teacher 
knowledge was found, in turn, to be strongly related to impact on practice, these features 
also had significant indirect effects on practice, student learning outcomes and teacher 
efficacy. 

As mentioned above, there were marked differences in the extent to which programs 
incorporated the opportunity to loam variables into their design. On a four-point scale the 
average score was about 3 for oontait focus and adive learning, 2,5 for follow-up and oollaboration 
and 1.5 for feedback , Not reported in Table 1 are the strong relationships between content 
focus, active learning, collaborative examination of student work and follow up across the 
four studies. These results suggest that programs with an emphasis on the subject matter 
that is being taught, how it is learned and how to teach it, tend to facilitate more active 
school based professional learning processes. As might be expected, this study found that 
programs that did build opportunities for follow up support into their design were more 
likely to provide opportunities for teachers to receive feedback as they tried out new skills. 
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Not included in this paper are findings related to control variables and structural features 
of programs, as set out in the model in Figure 1. In summary, these showed that for most 
Programs the level of school support influenced the extent of active learning, follow-up and 
feedbadr, and was related to the level of professional community activity resulting from 
programs. As these aspects of opportunity to learn are significantly related to the reported 
level of impact on knowledge, this indicates that the level of school support, as defined 
above, has substantial, though indirect effects on the extent to which program outcomes 
are achieved. Though not a design feature of most professional development programs in 
this study, the level of school support comes through the analysis as an important enabling 
condition with a significant shaping influence on the opportunities to learn that teachers 
experience. 

By and large the duration of programs (contact hours) did not have direct effects on the 
impact measures in these studies. However, it was related significantly to the reported level 
of content focus, active learning, collaboration and feedback (0.15). As these opportunity to 
learn features were found to have significant direct effects on reported impact levels, 
duration emerges as an important structural feature of programs. This study indicates the 
importance of giving program designers the time and resources that will enable them to 
incorporate these learning opportunities into their programs. 

These studies indicate that time span enabled programs to strengthen professional 
community activity, which, in turn, increased the likelihood that programs would have 
significant effects on teacher knowledge and practice. The time span of programs had a 
significant effect on the amount of time program participants reported they spent meeting 
informally with other participants in related activities, such as joint lesson planning and 
developing curriculum materials. These structural features of contact hours and time span 
both have substantial, though indirect, effects on program outcomes. 

The main message from these studies is that, among all the variables in the conceptual 
model (Fipre 1), the block of opportunity to learn or process variables had the largest 
effect on program outcomes. Of this set of variables, the ones with the most consistent 
effects were content focus (especially those that focused on how students learn the content 
and on methods to teach the content), active learning, and follow up. The fact that the 
influence of active learning carries through to influence teacher practices and teacher 
efficacy, net of the effects of other variables in the model, is particularly noteworthy. It 
suggests that this feature is having a pervasive and generative influence on factors that 
increase teachers' confidence and ability to meet student needs than making specific 
changes in practice alone. 


Discussion 

This study indicates that the most effective programs, in terms of reported impact, had 
profiles consistent with research on effective professional development (Hawley & Valh, 
1999). They were rated highly by teachers across all five opportunity to loam measures in the 
conceptual model (Fipre 1). They provided opportunities for teachers to focus on what 
students were to learn and how to deal with the problems students may have in learning 
that subject matter. They focused on research- based knowledge about student learning of 
content. They included opportunities for teachers to examine student work collaboratively 
- and in relation to standards for what the students in guestion should know and be able to 
do. They led teachers to actively reflect on their practice and compare it with high 
standards for professional practice. They engaged them in identifying what they needed to 
learn, and in planning the learning experiences that would help them meet those needs. 
They provided time for teachers to test new teaching methods and to receive follow-up 
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support and coaching in their classrooms as they faced problems of implementing changes. 
They included activities that led teachers to deprivatise their practice and gain feedback 
about their teaching from colleapes. 

This study is similar in methodology to the study by Caret et al. (2001). There are some 
differences in our model, reflecting the different context for the professional development 
programs included in the study. In the AGQTP, programs had to target, for example, 
experienced primary and secondary school teachers (10+ years) and areas such as literacy, 
numeracy, science and technology. Individual professional development programs, within 
these Programs, generally had to go through an extensive period of development and 
guality control mechanisms and, conseguently, were generally rated highly by teachers in 
terms of the impact measures. Our analysis was restricted to programs for which we had 
responses from at least ten teachers - and teachers were not surveyed until at least three 
months had elapsed since the program. 

Caret's study draws on their evaluation of the Eisenhower Professional Development 
Program - directed specifically at mathematics and science teachers. In selecting programs, 
they drew on a national prob^ility sample of school districts and others who had received 
Eisenhower funds. Their survey is based on a nationally representative sample of about 
1500 teachers who had attended these activities. They sub sampled two teachers from each 
activity. 

By and large the findings from the two studies are similar as well. Content focus and active 
learning have a significant impact on knowledge in both studies. The ACE R study includes 
separate measures for follow up, feedback and collaborative examination of student work, 
but the Caret study includes these in its overall index of active learning. We did not 
include a measure like G aret's ooherence, as it did not seem applicable to the Australian 
context where programs would not receive funding unless they were consistent with 
Commonwealth objectives and state- level curriculum standards. However, we did include 
a measure of the extent to which a professional development program facilitated the 
development of professional community at the school level as part of its strategy. This 
turned out to be a significant mediating variable in the ACER study. 

The strong relationship between content focus and reported impact on practice has been 
noted. This supports findings from Joyce and Showers (1982), Cohen and Hill (200), and 
reviews by Kennedy (1998) and Hawley and Valli (1999); that a strong knowledge base and 
a clear theoretical rationale grounded in research are necessary conditions for effective 
programs. An increased sense of teacher efficacy is, not surprisingly, dependent on the 
extent to which teachers think their practises have improved (increased competence) and 
evidence that student- learning outcomes have improved as a result. The strongest influence 
on teachers' reported levels of impact on efficacy in all four studies was the extent to which 
teachers saw that a program had had an impact on their students' learning outcomes. This 
finding echoes earlier research by Gusky (1985), which found that the more effective 
strategy is to ask teachers to try out new practices and see the effects on their students, 
rather than trying to change attitudes first in the hope that this will lead to change in 
practice. Programs that model effective practice and invite teachers to try them out tend to 
be more successful than programs that devote resources primarily to changing attitudes 
first. 

The opportunity b learn variables that appeared to have had the least influence in this study 
were feedback and collaborative ex amination of student work , despite strong evidence for their 
importance in other research studies. However, as reported above, few program designers 
in this study built opportunities for feedback about practice into their strategies. T eachers 
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indicated that opportunities to receive feedback as they tried new practices in the classroom 
were rare. Consequently there was little variation across programs in this variable and the 
study was not able to provide a fair test of the effect that feedback could have on teachers' 
practice. Similarly, few programs provided opportunities for collaborative examination of 
student work. Although this study does not provide a convincing case, it would not be 
appropriate to assume these two features were not important, given other research (Hawley 
& Valli, 1999). This study does indicate that program designers may not be incorporating 
these features to the extent that is needed to support implementation and sustained change 
in practice. 

The relative success of programs also depended on the extent to which programs were 
extended in time, and planned so that they included activities that strengthened interaction 
and collaboration in the school - the level of professional community activities. They 
linked to other programs in the participants' schools designed to improve learning. The fact 
that both span of time and contact hours show significant, though independent, effects in 
this analysis indicates that both aspects of duration are important in the design of effective 
professional learning activities. This finding is consistent with the research of G aret et al. 
(2001) who claim that, "Professional development is likely to be of higher quality if it is 
both sustained over time and involves a substantial number of hours" (p. 933). 

0 ther clear messages come through this analysis across the Programs. The role of follow-up 
is noteworthy. The level of follow up was found to increase significantly the extent to 
which teachers reported a sense of increased knowledge, perhaps reflecting the critical role 
that 'at the elbow' coaching and support in classrooms plays in learning new skills and 
putting them into practice. This kind of support was built into the more effective 
programs in our study. 

The findings from this study are also consistent with long standing research findings about 
the importance of school context. The pre-existing level of support for professional 
development in a school has a significant indirect effect on the outcomes of programs. It 
follows from this research that is not enough to provide well- designed professional 
development programs from outside the school. Policy makers and school administrators 
need to give equal attention to building the conditions that will enable schools to provide 
fertile ground for professional learning on an ongoing basis and as a routine part of the job. 
This study indicates that a substantial level of professional community is vM to significant 
change. The key ingredients here are time to think, analyse and talk about the specifics of 
what is going on in classrooms and what students are doing and learning. Effective school 
administrators in strong professional communities expect evidence of professional 
development and act in ways that demonstrate they value teacher learning. 

The findings from this cross-program analysis reflect findings from other research on 
professional development and challenges that policy makers have faced for many years. 
These studies often find a considerable gap between the conditions that research indicates 
are optimal for professional learning and those that are provided. The capacity of policy 
makers to provide funding sufficient to ensure their incorporation into most professional 
development programs can be constrained depending on the level of available funding for 
professional development. While the research has long indicated that there are no short 
cuts to significant and sustainable change at the classroom level, short cuts often have to be 
taken. However, money spread thinly, when it comes to professional development is 
unlikely to produce significant change. Where significant change is sought, it may be wiser 
to involvefewer teachers than produce less significant change among many. 
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Not only policy makers, but also professional development providers face a challenge in 
designing programs consistent with research in this field. This cross- Program study 
reinforces the central importance of building follow up support into the design of 
professional development programs. However, this feature of professional development is 
labour intensive and time consuming and, conseguently, expensive. 

The same applies to feedback , Perhaps one of the most significant findings in this study 
across the eighty programs was how rarely designers built in opportunities for feedback and 
coaching in the workplace, despite research on their centrality to learning new and complex 
skills. Reflection on practice and the development of understanding about the change is 
unlikely to be optim^ without sources of timely and insightful feedback on what one is 
doing. Ensuring opportunities for every teacher to receive 'at the elbow' support and 
coaching during the difficult phase of implementing significant change in the classroom is a 
feature of effective programs. We suspect that one of the most important challenges 
emerging from this study, especially for people with responsibility for ensuring professional 
development budgets are spent wisely, is to develop funding pidehnes that ensure that 
programs desipers include peater opportunities for participants to benefit from rich and 
freguent feedback. 

Similarly, time to bring teachers together in the workplace to examine student work and to 
provide opportunities for feedback needs to be built into current conceptions of teachers' 
work. These studies indicate that teachers' work needs to be organised on the assumption 
that it is professional work; that is, work that reguires an appropriate balance between up- 
front practice and back room collegial analysis and reflection on practice in the light of 
standards for student learning and professional practice. 

Limitations 

As described above, our approach to evaluation was based primarily on teacher self-report 
data. G iven the time frame and the level of resources usually allocated to evaluations of 
professional development propams, there is often little opportunity to gather first-hand 
evidence about changes in teacher knowledge, practice, efficacy and students' learning 
outcomes. We are more confident about the measures of impact on practice than impact 
on student learning outcomes. The graphs in Appendix C show peater variation for impact 
on practice than impact on student outcomes. However, recent studies (e.g. Mayer, 2001) 
indicate that it is reasonable, in certain circumstances, to place a certain level of confidence 
in surveys that rely on teachers' reports about their practice. Reliability of these self-report 
data increases as outcome measures become more specific, as those used here in the ACER 
approach. Also, teachers are not reluctant to speak their minds frankly when it comes to 
assessing the value of professional development propams. There is little reason to think 
that their responses might be biased one way or another, or any desire to please, especially 
when, in studies such as the above, they are contacted several months at least after the 
propams have finished. 


Note 

Paper presented to the Annual Meeting of the American Educational Research Association, 
San D iego, April 12- 16, 2004, The projects reported here were funded by the Australian 
G ovemment D epartment of Education, Science and Training, the Catholic Education 
Commission of V ictoria, the N SW Q uality T eacher Propamme, and the N orthem 
T erritoiy D epartment of E ducation. 
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Appendix C 


PROJECT "G": PROFILE 
T eaching and Learning Literacy 

BACKGROUND 

Program "G " was developed in response to K - 6 classroom teachers who were asking for 
support in their teaching of hteracy. The skills they had identified they needed were similar 
to those provided to Reading Reoovay teachers, but adapted for regular classroom teaching. A 
trial program was developed and evaluated by a group of 14 teachers over 12 months, before 
going to scale. The program advocated a functional approach to learning about literacy 
teaching and used a approach adapted from the Reading Recovery training model. 

STRATEGY 

The professional learning strategy for teachers in this project was the provision of 10 
workshops over one year. Using an 'immersion in inquiiy-case study' model, teachers 
examined student work to develop standards and continua in reading and writing. T eacher 
participants joined the course with at least one other teacher from their school to form 
school partnerships in learning. This created a learning community of participants, built 
over the year and continuing through ongoing contact. It also was designed to contribute to 
professional development in the area of leadership, with teachers working in schools as 
leaders; in addition, some teachers became facihtators of groups within each phase of the 
course. 0 pportunity was provided to the participants to negotiate the agenda. 

Teachers were required to use videos to capture their teaching for observation, discussion 
and reflection. This entailed coaching and mentoring, with school visits by the facilitator 
and joint video watching to set goals, guide practice and evaluate practice. The facihtator 
communicated through the training workshops, teleconferences and through intranet 
discussion. Technology schools, in particular, were visited for review and inspiration 
regarding the diverse ways technology was used to support learning. Participants were 
expected to share their experience and learning from the course with their non- participant 
colleagues in their schools. The course aimed to reduce the notion and practice of isolated 
learning and to develop school culture change. 

A number of key beliefs underpinned this project. These included the need to develop and 
provide professional development according to the principles of effective professional 
development - for example, that both the participants and the school make the course a 
priority; that there is a bdance between theory and practice; that learning is both formative 
and reflective, and that teachers can work with and learn from their colleagues and their 
students in trialling new learning. 




Appendix D 

Program B; Relationship between background variables, structural features, opportunity to learn, professional community in 
the school, and teacher knowledge, teacher practice, student learning and teacher efficacy 
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